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The Education Endowment Foundation (EEF) 


The Education Endowment Foundation (EEF) is an independent grant-making charity dedicated to 
breaking the link between family income and educational achievement, ensuring that children from all 
backgrounds can fulfil their potential and make the most of their talents. 


We aim to raise the attainment of children facing disadvantage by: 


e Identifying promising educational innovations that address the needs of disadvantaged 
children in primary and secondary schools in England; 


e Evaluating these innovations to extend and secure the evidence on what works and can be 
made to work at scale; 


e Encouraging schools, government, charities, and others to apply evidence and adopt 
innovations found to be effective. 


The EEF was founded in 2011 by lead charity The Sutton Trust, in partnership with Impetus Trust 
(now part of Impetus — The Private Equity Foundation), with a £125m grant from the Department for 
Education. With investment and fundraising income, the EEF intends to award as much as £200m by 
2026. Together, the EEF and Sutton Trust are the Government-designated What Works Centre for 
Improving Education Outcomes for School-Aged Children. 
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Literacy Catch-up Round 


This study was funded by the Education Endowment Foundation as one of 23 projects funded through 
a themed funding round on literacy catch-up at the primary-secondary transition. Projects funded 
within this round aimed to identify effective ways to support pupils who do not achieve Level 4 in 
English by the end of Key Stage 2. The project was one of three funded with a particular focus on 
writing. 


For more information please contact: 


Robbie Coleman 
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Research and Communications Manager 
Education Endowment Foundation E N e| O W mM e N t 


9th Floor, Millbank Tower 
21-24 Millbank 
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p: 020 7802 1679 
e: robbie.coleman@eefoundation.org.uk 
w: www.educationendowmentfoundation.org.uk 
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David Torgerson 


Professor David Torgerson, Director of the York Trials Unit, joined the Centre for Health Economics at 
the University of York in 1995 and became the Director of the York Trials Unit in 2002. Originally a 
health economist, he is a trial methodologist and has published widely on the design and conduct of 
randomised controlled trials including the book Designing Randomised Controlled Trials in Health, 
Education and the Social Sciences (Palgrave Macmillan, 2008). David has over 200 peer-reviewed 
publications. 
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Carole Torgerson 


After taking degrees in English Literature and Modern English Poetry from the Universities of York and 
Stirling, Carole Torgerson taught English in a wide variety of schools in England and Scotland. She 
undertook her doctorate in design issues in experimental research at the University of Sheffield before 
starting her academic career. She was Reader in Evidence-based Education at the University of York, 
before moving to The University of Birmingham as Professor of Experimental Design and then to 
Durham University where she is Professor of Education. 
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Executive summary 


The project 


The Discover Summer School was a four-week programme which aimed to improve the reading and 
writing skills of children during the summer between Year 6 and Year 7. The programme was targeted 
at pupils who had been predicted to achieve below Level 4b in English by the end of Key Stage 2. 


The programme involved morning workshops focused on writing and poetry delivered by trained 
teachers, professional authors and poets, and enrichment activities provided in a four-week summer 
school. Pupils meeting the eligibility criteria for the study, and due to transition from Year 6 to Year 7 
in 2013, were recruited through 29 primary schools in proximity to the Discover Children’s Story 
Centre in Stratford, London in July and August 2013. The study involved 124 pupils from 29 local 
schools, of whom 76 were randomly selected to attend the summer school; the remaining 48 formed a 
comparison group. 


The evaluation was funded by the Education Endowment Foundation as one of 23 projects focused on 
literacy catch-up at the primary-secondary transition. It was one of two summer school programmes 
funded, and one of three programmes with a particular focus on writing. 


What impact did it have? 


The evaluation sought to compare the literacy skills of pupils who attended the summer school with 
similar pupils who did not attend the summer school. The indicative effect sizes detected were 0.24 in 
writing and 0.21 in reading. This can be envisaged as saying that pupils who participated in the project 
made approximately three additional months’ progress compared to similar pupils who did not 
participate. However, the effect sizes were not statistically significant, which means that the 
differences between the pupils who attended the summer school and those who did not could have 
been down to chance. 


The conclusions which can be drawn from this study are severely limited due to the small number of 
families who agreed to participate in the trial, and to subsequent problems with testing, detailed further 
below. The low pupil numbers also meant that it was not possible to conduct a separate analysis of 
the impact of the programme on students eligible for free school meals. 


Though the evaluation was primarily focused on measuring attainment, the observations which took 
place suggested that most children were engaged by and enjoyed the summer school. Staff were 
supportive and children were encouraged to think creatively about their writing. The interactive 
approach in poetry sessions in particular appeared to motivate pupils. 


All pupils (Writing) 124 +0.24 +3 -0.42 to 0.92 


All pupils (Reading) 124 +0.21 +3 -0.41 to 0.83 


*Effect sizes with confidence intervals that pass through 0 are not ‘statistically significant’, 
which means the difference could be due to chance. 


*“For more information about evidence ratings, see Appendix 11. 
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How secure is this finding? 


This evaluation was set up as an efficacy trial. Efficacy trials seek to test evaluations in the best 
possible conditions, but they do not seek to demonstrate that the findings hold at scale in all types of 
schools. 


An individually randomised controlled trial design was employed to compare outcomes for pupils 
attending the summer school with outcomes for pupils in a ‘business as usual’ comparison group who 
did not attend. Eligible and consenting pupils were randomised on an individual basis to either the 
intervention or comparison group. At the end of the intervention period all trial pupils were asked to 
complete the Progress in English test developed by GL Assessment, with the primary outcome 
measure being based on two extended writing tasks. Pupils in the comparison group were offered 
additional reading and writing support delivered on one Saturday in the autumn term of 2013. 


However, the intervention and evaluation suffered a number of substantial setbacks. The initial 
recruitment target of 250 pupils was not achieved by Discover, with only 124 eligible pupils agreeing to 
take part in the intervention and evaluation. The length of the summer school and the timing of the 
recruitment phase made recruitment more difficult: many parents had already booked holidays or 
made alternative plans for the summer by the time they were invited to participate. In addition, a large 
proportion of pupils in both the intervention and comparison groups did not complete the tests. These 
problems made it impossible to detect a statistically significant effect size and increased the possibility 
of bias as the pupils who sat the tests may have been systematically different from those who did not. 
As a result, the evaluation findings are insecure. 


How much does it cost? 


The cost of the programme is estimated at £1,750 per pupil. This estimate includes venue hire, food 
and travel (estimated at £635 per pupil), direct salary costs of staff (£773), promotion and contingency 
(£94) and management and overheads (£288). This estimate is based on 125 pupils attending a 
summer school on a single site. 


1. The evaluation was unable to provide a secure estimate of the programme’s impact on reading or writing 
attainment, due to the low number of pupils who took part and the problems with testing. 


2. The programme was relatively expensive compared to other literacy catch-up approaches delivered in the 
normal school year. 


3. Given the relative cost of the programme, even if the indicative impact which was detected had been secure, as 
a way of improving academic outcomes other approaches are likely to be more cost-effective. 


4. Participating pupils enjoyed the programme and were engaged by the literacy workshops and the enrichment 
activities. 


5. The specific challenges of pupil recruitment, attendance and test completion experienced should be considered 
prior to undertaking any further evaluations of summer schools. 
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Introduction 


The Discover Summer School consisted of a range of literacy-based and enrichment activities 
delivered to children over a four-week period in the summer between the end of Year 6 and the start of 
Year 7. Further information about the intervention can be found in the Methods section of this report. 


Background evidence 


In the ‘Moving English Forward’ report, Ofsted highlighted persistent problems with attainment in 
English at Key Stage 2 (KS2) level, including weaker attainment in English amongst pupils who are 
eligible for free school meals (FSM) (Ofsted, 2012). Attainment overall has remained static, with 79% 
of pupils achieving Level 4 or above in 2005 and a slight rise to 82% in 2011; however, standards in 
writing have reduced over time (Ofsted, 2012). 


There has been increasing focus over recent years to improve the ‘transition’ period for pupils moving 
from the final year of primary school (Year 6) to the first year of secondary school (Year 7). In a 
review exploring the effects of transition, McGee et al. (2004) highlight a breadth of international 
literature demonstrating a decline in academic attainment following transition. Furthermore, research 
which surveyed children about their experiences of transition from primary to secondary schools 
suggests that children with low socioeconomic status may have less positive transitions (Evangelou et 
al., 2008). 


Summer school programmes have been identified as a potential means through which pupils’ 
transition to secondary schools can be eased. In 2011, the UK government provided £50m funding to 
support a summer schools programme for disadvantaged pupils, specifically those eligible for FSM 
and those cared for by local authorities for periods of six months or more, which aimed to make the 
transition from primary to secondary school more successful (Day et al., 2013). Literacy-focused 
summer schools for Year 6 children have also been identified recently, alongside other activities, as a 
strategy for English teachers to use to ease this transition (Ofsted, 2012). 


Despite this investment in summer schools over recent years, there is a lack of evidence from 
rigorously designed evaluations to establish their effect on pupil attainment. Sainsbury et al. (1998) 
and Day et al. (2013) describe the findings of cohort studies which surveyed pupils before and after 
attending summer schools. In 2012, the University of York and Durham University were funded by the 
Education Endowment Foundation (EEF) to evaluate independently the Discover Summer School as 
delivered by the intervention developers based at the Discover Children’s Story Centre, a charitable 
organisation based in Stratford, London. 


The evaluation was funded by the Education Endowment Foundation as one of 23 projects focused on 
literacy catch-up at the primary-secondary transition. It was one of two summer school programmes 
funded, and one of three programmes with a particular focus on writing. 


Evaluation objectives 


The objective of the independent evaluation was to test the effectiveness of the Discover Summer 
School compared with a ‘business as usual’ control group, on the reading and writing abilities of 
participating children. 
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Project team 


The evaluation team, jointly comprising staff from Durham University and the University of York, was 
responsible for the design, conduct, analysis and reporting of the independent evaluation. 


We thank Dr Taeko Becque for her contribution to writing the statistical analysis plan used in this 
evaluation. 


Implementation Team 


Staff at Discover Children’s Story Centre were responsible for school recruitment and ongoing 
relationships with schools, informing parents and pupils, intervention development (including writing a 
detailed description of the intervention to allow others, if necessary, to be able to replicate the 
intervention in other areas), baseline data collection and intervention delivery. 


Ethical review 


Ethical approval was obtained for this study from Durham University School of Education Ethics 
Committee. Opt-in consent was sought from the parents of all participants. 
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Trial design 


An individually randomised controlled trial design was employed to compare outcomes for pupils 
attending the Discover Summer School with outcomes for pupils in a ‘business as usual’ control group 
who did not attend the summer school. Eligible and consenting pupils were randomised on an 
individual basis to either the intervention or control group. Further details of these processes and the 
interventions are found below. 


The trial was designed, conducted and reported to CONSORT standards (Altman et al., 2011) in order 
to minimise all potential threats to internal validity, such as selection bias and a range of post- 
randomisation biases (Cook and Campbell, 1969; Shadish, Cook and Campbell, 2002; Torgerson and 
Torgerson, 2008). As a result, unbiased estimates of the impact of the intervention should be provided 
if attrition and missing data are relatively low. Due to data restrictions relating to the eventual small 
sample size in this trial, analyses presented differ to those planned in the protocol (Appendix 1) and 
statistical analysis plan (Appendix 2). 


Recruitment 


School recruitment took place during the development phase of the intervention (between March and 
May 2013) and was assisted through information leaflets for schools (Appendix 3) that were 
developed by the evaluation team. Discover recruited primary schools that fed into local secondary 
schools that had agreed to undertake outcome testing in the autumn term of 2013. Schools were 
asked to sign an ‘Agreement to Participate Form’ (Appendix 4 and 5) to ensure they agreed to all trial- 
related procedures. 


Parental information sessions were held by the implementation team (Discover) to raise awareness of 
the trial amongst parents of eligible children. Parents of eligible pupils were provided with information 
letters about the study, explaining the nature of randomisation to either control or intervention groups, 
the use of outcome testing and pupil data (Appendix 6). Parents were asked to return consent forms to 
confirm that they were willing for their child to take part in the study (Appendix 7) and following this, 
participating primary schools then shared consenting pupils’ data with the evaluation team (including 
pupil name, unique pupil number (UPN), date of birth (DOB), free school meals status (FSM), and 
predicted Key Stage 2 (KS2) English based on teacher assessment). 


Eligibility 


School inclusion criteria: Primary and secondary schools were eligible to take part in the trial if they 
agreed to all trial procedures, including: identifying pupils meeting the inclusion criteria for the study, 
informing parents, provision of pupil data and implementation of the outcome testing. 


Pupil inclusion criteria: Pupils were eligible if they were going to be attending a participating 
secondary school, if they were predicted to achieve Level 3c, Level 3b, Level 3a or Level 4c in English 
by the end of Key Stage 2 (based on teacher assessment) and if their parent/guardian consented for 
them to take part. 


Education Endowment Foundation 6 


Methodology 


School exclusion criteria: Schools were excluded from participating in the trial if they did not agree 
to all points listed in the ‘Agreement to Participate Form’ and primary schools were excluded if they did 
not feed into one of the secondary schools that had agreed to administer the outcome testing. 


Pupil exclusion criteria: Pupils were excluded from individual randomisation if they were not 
attending a participating secondary school or were expected to achieve below Level 3 or above Level 
4c. Those predicted to achieve below Level 3 were excluded from testing as it was thought the post- 
testing could have caused undue anxiety. 


Intervention 


The Discover Summer School was delivered through a four-week summer programme for children in 
transition from Year 6 to Year 7. The intervention aimed to improve children’s reading and writing 
skills, specifically those who were less able writers. The programme included a variety of workshops, 
including poetry and literacy sessions, in which children had the opportunity to engage with 
professional poets and writers. A range of enrichment activities was also provided including drama, 
sports activities and trips around London, such as West End performances and visits to the Olympic 
Park. The summer school was delivered by trained ‘Story Builders’ from Discover, as well as teachers 
and volunteers including trainee teachers, and was designed to follow the patterns of a usual school 
week (9am-3pm, Monday to Friday, with a breakfast club open from 8.30am). In order to encourage 
greater attendance, free transport arrangements were provided from multiple collection and drop-off 
points for pupils attending the summer school. 


The control group was a ‘business as usual’ group, with pupils undertaking usual activities over the 
summer period. In this respect, this evaluation examines the potential efficacy of a supplementary 
intervention delivered in the summer holiday between Year 6 and Year 7 (i.e. supplementary to normal 
school experience) rather than an alternative core intervention delivered during the normal school day. 
It is possible that the control condition may have involved attendance at another summer school in the 
area hosted, for example, by the receiving secondary school. As an incentive to increase recruitment 
and avoid disappointment, the control group was offered a secondary school-based reading and 
writing intervention delivered on one Saturday in the autumn term 2013. This secondary school-based 
intervention did not form part of the evaluation as it took place after the outcome testing. 
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Outcomes 


The Progress in English (PiE) 11: Second Edition Long Form (LF) Test, GL Assessment, was the main 
literacy outcome. The test included both narrative and non-narrative exercises and assessed both 
reading and writing skills including areas such as spelling, grammar and comprehension. The 
Progress in English test was the only test available to the evaluation team (in order to comply with 
EEF testing policy) which included a writing component. Tests were marked by GL Assessment blind 
to allocation (i.e. markers did not know whether test papers were from intervention or control pupils). 


The pre-test was pupils’ performance in Key Stage 2 English at the end of Year 6, obtained through 
the national pupil database (NPD). 


Primary outcomes 


The primary outcome was extended writing score which referred to the combined raw score on the two 
extended writing tasks (Exercises 5 and 6) from the PiE 11 LF. Exercise 5 had a maximum of 20 
marks and involved writing a persuasive letter; Exercise 6 had a maximum of 12 marks and assessed 
informative writing. Overall, the extended writing task score could therefore range from 0 to 32, with a 
higher score representing higher attainment. 


Secondary outcome 


Secondary outcomes were based on pupils’ reading, spelling and grammar scores. Reading score 
was based on the combined raw score on the reading tasks 3, 4, 3x and 4x. Exercise 3 (comprising 
Exercises 3 and 3x) had a maximum of 19 marks and assessed reading comprehension of a narrative. 
Exercise 4 (comprising Exercises 4 and 4x) had a maximum of 13 marks and assessed non-narrative 
reading comprehension. Overall, reading score could therefore range from 0 to 32, with a higher score 
representing better attainment. 


Spelling and grammar score was based on the combined raw score on the spelling and grammar 
tasks (Exercises 1 and 2). Exercise 1 had a maximum of 10 marks and assessed spelling; Exercise 2 
also had a maximum of 10 marks and assessed grammar. This means the spelling and grammar 
score combined could range from 0 to 20, with a higher score representing higher attainment. 


Delivery of outcomes 


Teachers were asked to deliver the outcome tests under ‘exam’ conditions. They were not informed of 
the group allocation of the children, although it is possible that children may have informed teachers if 
they attended the summer school or not. 
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Sample size 


This trial aimed to recruit 250 pupils. To estimate the differences that we could estimate with 80% 
power (and a two-sided significance level of 0.05, i.e., 26 = 0.05), we used the assumption that there 
would be a pre- and post-test correlation of 0.70 (i.e. the outcome test would correlate with the Key 
Stage 2 English result with a correlation of 0.70). This means that there would have been the 
equivalent sample size of 492 if no pre-test had been conducted. Allowing for a 10% attrition rate gave 
an effective sample size to 442 pupils. This gave 80% power to show a difference of 0.27 standard 
deviations in writing score between the intervention and control groups, should a difference of this 
magnitude exist. 


Randomisation 


Pupils were individually randomised to either the intervention or control group using an unequal 
allocation ratio of 3:2 (intervention:control). Randomisation was stratified by secondary school for 
practical reasons and a fixed block size of 10 was employed. The trial statistician, based at York Trials 
Unit, undertook the randomisation in Stata® version 12 (Stata Corporation, College Station, Texas, 
USA) after the pupil baseline data had been received. The evaluation team provided allocation 
information to the implementation team, who provided this information to parents of pupils and 
disseminated further information about the summer school to those in the intervention arm. 


Two sets of twins were included in this trial. For practical reasons, it was requested that each of the 
twins within a pair be assigned to the same trial arm. All pupils were randomised separately: in the 
case of the twins, each pair received the allocation assigned to the ‘first’ twin as defined by the 
smallest trial ID number. 


It was planned that pupils allocated to the intervention would be randomly assigned to one of five 
teaching groups such that the groups were of identical size. Following consultation with the 
implementation team about this, and in order to be pragmatic and reflect what would happen in usual 
practice, pupils were divided into four groups by teachers for the following reasons: 1) several pupils 
fasting for Ramadan were grouped together; 2) pupils attending the same secondary schools in 
autumn term were grouped together; 3) pupils could be moved into different groups should 
disagreements arise between pupils in the same group. 


Analysis 


Analysis was conducted in Stata® version 13 (Stata Corporation, College Station, Texas, USA) using 
the principles of intention-to-treat, meaning that all pupils were analysed in the group to which they 
were randomised irrespective of whether or not they actually received the intervention and irrespective 
of implementation fidelity. Statistical significance was assessed at the 5% level, unless otherwise 
stated. 


Effect sizes were calculated using the Hedges’ g method and are presented alongside 95% 
confidence intervals. 


Education Endowment Foundation 9 


Methodology 


Primary analysis 


The difference between pupils receiving the Discover four-week summer programme and those pupils 
in the ‘business as usual’ control group was compared using regression analysis with extended writing 
post-test result as the response variable. It was planned that group allocation, gender, FSM status, 
EAL status and predicted KS2 score would be included as covariates in the model. Adjustment was 
also planned for secondary school which was used as a Stratification factor during randomisation. 
Given the low levels of post-test completion and the resulting limited number of complete case 
observations available for inclusion in this analysis, as well as the low numbers of pupils per 
secondary school, it was not possible to include the stratification factor in the primary analysis (EMEA, 
2003) and the fitted model only adjusted for predicted KS2 level. 


Secondary analyses 


An analogous method to the primary analysis was used to assess differences between the 
intervention and control groups in terms of the secondary outcome of spelling and grammar. In relation 
to the secondary outcome of reading, there were not enough pupils fully completing the reading 
outcome exercises to conduct any statistical analyses. 


A subgroup analysis to investigate the effect of the intervention on the writing skills of pupils who were 
eligible for FSM was planned but was not conducted as the low numbers of complete cases relating to 
the primary outcome of extended writing score did not allow for the inclusion of an interaction term 
within the model. 


A further subgroup analysis to assess the effect of the four teaching groups was also planned but not 
possible. 


Patterns of missing data in the post-test result data were explored. Multiple imputation was not 
deemed appropriate, particularly due to the large amounts of missing data. 


Post-hoc analysis 


It was not possible to adjust for FSM status, EAL status and gender in the primary analysis. As a 
result, three regression analyses were conducted adjusting for predicted KS2 level and one of the 
three additional covariates one at a time. 


Due to the low level of post-test completion and the resulting limited number of complete case 
observations, post-hoc analyses were requested by the funding body. Based on the data, two such 
analyses were conducted. Both used the regression approach of the planned primary analysis: the 
first used scores from Exercise 5 only as the response and the second used combined raw score on 
the two reading exercises from the short form of the test (Exercise 3 and 4) as the response. These 
analyses on part of the primary outcome and secondary outcome of reading were requested due to 
the number of pupils who were partially missing primary outcome data. Given that the amount of data 
was still limited it was again not possible to include the stratification factor of school in the primary 
analysis (EMEA, 2003). 
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Attendance 


Summary statistics of the number of sessions attended were produced to describe fidelity to the 
intervention. Reasons for non-attendance were collected, where possible, and are reported in the 
results section of this report. 


Process evaluation methods 


Non-participant observations were undertaken by a researcher independent to the intervention 
delivery (LJ) in order to observe the delivery of the Discover Summer School. Owing to the short 
timeframe for this study, in-depth process evaluation was not conducted and observations were 
primarily undertaken to gain greater insight into the nature of the intervention, in particular the delivery 
of literacy-based teaching sessions. Formal qualitative analysis of these observations was not 
conducted. 


University ethics approval was also sought for this part of the evaluation. Prior to commencing 
observations, the teachers were provided with an information leaflet about the observations and 
written consent was obtained prior to commencing any observations (Appendices 8 and 9). 
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Timeline 


School recruitment took place from February to May 2013, with pupil recruitment running concurrently 
to this. Baseline data for the trial were collected and pupils randomised in June 2013. The summer 
school took place from 29th July to 23rd August and pupils were tested within the first 3 weeks of 
return to school following the summer holidays. 


Recruitment and follow-up of participants 


Pupils from 29 primary schools were identified that met the inclusion criteria for the study and were 
also due to be attending participating secondary schools (N = 14) in the autumn term of 2013. 
Participating primary and secondary schools were all within a similar geographic location close to the 
Discover Children’s Story Centre in Stratford, East London. 


Consent was obtained from the parents or guardians of 186 pupils, however, due to changes in the 
secondary school choice for 62 pupils, a total of 124 pupils were eligible and randomised. Ascertaining 
which secondary school pupils planned to attend was particularly problematic as the study was 
located close to a number of potential schools, leading to late decisions and changes. Not all these 
schools had agreed to outcome testing (one of the requirements for pupil eligibility) resulting in a 
reduced pool of eligible pupils for this trial. For fairness, as these 62 pupils had already been informed 
of the study and offered the opportunity to take part in the summer school, these children were 
randomised (on a 3:2 basis) to either the intervention or control group. No further data were collected 
or used for these pupils in this evaluation. The CONSORT diagram in Figure 1 describes the flow of 
participants through this study. Throughout the results section, N is used to refer to the number of 
schools and n is used to refer to the number of pupils. 
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Figure 1. CONSORT flow diagram 


Consent Form Returned 


n= 166 


Randomised 
n= 124 


Intervention 
Allocated: n= 76 


Attended all sessions: n= 7 
Attended some sessions: n= 41 
Attended no sessions: n= 28 


Analysed [in primary analysis) 
n= 20 (26.3%) 


Baseline characteristics 


Table 1 shows characteristics of the 29 primary schools from which pupils were recruited to the trial. 
The mean school size was just under 600 pupils with a range between 250 and nearly 1,000 pupils. 
Just over one third of pupils on roll were eligible for FSM, which is considerably higher than the 
national average of 19.2% for 2013 (Department for Education, 2013). Over 40% had received FSM at 
some point in the last 6 years. The proportion of pupils with English as an additional language was 


high, with a mean of over 70%. 


Table 1. Characteristics of recruited schools 


Excluded due to not attending a 


participating secondary school 
n= 62 


Control 
Allocated: n= 48 


Received control n= 48 


Analysed (in primary analysis) 
n= 14 (29.2%) 


Mean (SD) 


578.9 (188.24) 


Med (Min, Max) 


534 (250, 993) 


Missing (%) 


0 (0.0) 


Education Endowment Foundation 


Percentage pupils eligible for FSM 


Mean (SD) 36.7 (12.4) 
Med (Min, Max) 34.8 (17.6, 68.2) 
Missing (%) 0 (0.0) 


Percentage pupils receiving FSM in the last 6 years 


Mean (SD) 43.5 (12.6) 
Med (Min, Max) 42.3 (24.8, 77) 
Missing (%) 0 (0.0) 


Percentage pupils with English as an additional language 


Mean (SD) 71.1 (16.6) 
Med (Min, Max) 73.2 (28.5, 96) 
Missing (%) 0 (0.0) 


Percentage pupils with SEN statement or on School Action 


Plus 
Mean (SD) 7.4 (2.77) 
Med (Min, Max) 6.2 (3.4, 11.8) 
Missing (%) 0 (0.0) 


Table 2 shows baseline pupil-level characteristics by allocated group (i.e. intervention and control), 
both as randomised and as analysed in the primary analysis. In relation to the demographic 
characteristics of age, FSM status, Pupil Premium (PP) status, and predicted KS2 writing level at 
baseline, proportions of pupils within each category were fairly similar between the intervention and 
control arms both as randomised and as analysed in the primary analysis, although numbers eligible 
for analysis were small due to missing data. The proportion of those eligible for inclusion in the primary 
analysis with English as an additional language was much higher in the control arm than in the 
intervention arm (92.9% compared with 65.0%). 
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Table 2. Baseline characteristics of participating schools and classes 


Intervention 


Control 


Intervention 


Control 


n = 76 (61.3%) 


n = 48 (38.7%) 


n = 20 (58.8%) 


n= 14 (41.2%) 


Age 
Mean (SD) 11.3 (0.32) 11.3 (0.38) 11.3 (0.30) 11.3 (0.33) 
Median 11.3 11.4 11.3 11.4 
Minimum and 10.8, 12.4 10.0, 12.1 10.8, 11.7 10.9, 11.9 
maximum 
Gender 
Male (%) 43 (56.6) 29 (60.4) 12 (60.0) 7 (50.0) 
Female (%) 33 (43.4) 19 (39.6) 8 (40.0) 7 (50.0) 
Missing (%) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 
FSM 
Eligible (%) 32 (42.1) 27 (56.3) 8 (40.0) 9 (64.3) 
Not eligible (%) 44 (57.9) 21 (43.8) 12 (60.0) 5 (35.7) 
Missing (%) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 
Pupil premium 
Eligible (%) 31 (40.8) 23 (47.9) 10 (50.0) 8 (57.1) 
Not eligible (%) 45 (59.2) 25 (52.1) 10 (50.0) 6 (43.9) 
Missing (%) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 
English as an 
additional 
language 
EAL (%) 42 (56.0) 33 (68.8) 13 (65.0) 13 (92.9) 
Non-EAL (%) 33 (44.0) 15 (31.3) 6 (68.4) 1A) 
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Missing (%) 1 (0.8) 0 (0.0) 1 (5.0) 0 (0.0) 
Predicted KS2 
writing level 
Level 3c (%) 8 (10.5) 3 (6.3) 2 (10.0) 2 (14.3) 
Level 3b (%) 8 (10.5) 8 (16.7) 3 (15.0) 1 (7.1) 
Level 3a (%) 21 (27.6) 10 (20.8) 5 (25.0) 3 (21.4) 
Level 4c (%) 38 (50.0) 25 (52.1) 9 (45.0) 8 (57.1) 
Level 4b (%) 0 (0.0) 1 (2.1) 0 (0.0) 0 (0.0) 
Level 4a (%) 1 (1.3) 1 (2.1) 1 (5.0) 0 (0.0) 
Missing (%) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 


Outcomes and analysis 


Raw, unadjusted mean post-test scores are presented in Table 3 by trial arm. Extended writing scores 
were fairly similar in both allocated groups at 18.3 out of 32 marks (SD 4.34) in the intervention group 
and 17.3 marks (SD 2.76) in the control group. Proportions of those completing the extended writing 
questions were low but similar in both allocated arms; overall only 27.4% of pupils completed the 
primary outcome. In relation to the secondary outcomes of reading, and spelling and grammar, mean 
scores were also similar between the arms, as were proportions completing. 


Table 3. Unadjusted average scores for the intervention and control groups 


n= 76 n= 48 n=124 
Primary outcome 
Extended writing score n=20 n=14 n=34 
Mean (SD) 18.3 (4.34) 17.3 (2.76) 17.9 (3.75) 
Med (Min, Max) 17.5 (8, 27) 16.5 (13,23) 17 (8, 27) 
Completely missing (%) 44 (57.9) 27 (56.3) 71 (57.3) 
Partially missing (%) 12 (15.8) 7 (14.6) 19 (15.3) 
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Complete (%) 20 (26.3) 14 (29.2) 34 (27.4) 
Secondary outcomes 
Reading score n=24 n=16 n=40 
Mean (SD) 12.5 (3.86) 11.8 (4.04) 12.2 (3.90) 
Med (Min, Max) 13 (3, 20) 11 (6, 19) 12 (8, 20) 
Completely missing (%) 26 (34.2) 20 (41.7) 46 (37.1) 
Partially missing (%) 26 (34.2) 12 (9.7) 38 (30.6) 
Complete (%) 24 (31.6) 16 (33.3) 40 (32.3) 
Spelling and grammar score n=44 n=25 n=69 
Mean (SD) 8.0 (4.11) 7.2 (3.59) 7.7 (3.92) 
Med (Min, Max) 8 (0, 17) 6 (0, 13) 7 (0, 17) 
Completely missing (%) 27 (35.5) 20 (41.7) 47 (37.9) 
Partially missing (%) 5 (6.6) 3 (6.3) 8 (6.5) 
Complete (%) 44 (57.9) 25 (52.1) 69 (55.6) 


Post-test data were missing for 71 pupils (57.3%). Just over two fifths (43.6%) of pupils did not 
complete any questions from the long form part of the paper. All schools attempted the long form part 
of the test, however all pupils at one school were completely missing scores for the long form 
exercises. Due to the small pupil numbers at this school, it is not possible to say with certainty that this 
school did not set pupils the long form of the test (more details in Appendix 10). 


Only 27.4% of pupils completed the extended writing exercises in full. Of the 15.3% of individuals who 
partially completed the extended writing section of the paper, all were missing responses to Exercise 
6. Higher proportions of those with English as an additional language had an extended writing score 
than those with English as a first language. Proportions completing and not completing the test were 
similar irrespective of FSM status, gender, or whether the pupil was predicted a Level 3 or a Level 4. 


There are a number of possible factors which may have contributed to the low rates of test completion. 
In some cases there was confusion over which pupils were required to sit tests, and it is possible that 
others may have experienced difficulty in testing only a handful of students or in finding adequate time 
for children to sit the tests, resulting in incomplete tests and missing data. For some schools, delivery 
of the correct tests was delayed, which may have reduced engagement with testing in some cases. 


Just under one third of pupils attempted all reading exercises and hence had a reading score. Two of 
the reading exercises were part of the long form of the test; whilst 75 pupils attempted Exercises 3 and 
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4 (the short form reading questions), only 40 of these also attempted both Exercise 3x and Exercise 
4x (the long form reading questions). Just over half of the pupils (55.6%) attempted both Exercises 1 
and 2 and hence had a spelling and grammar score. 


Exercise 3 was the most frequently completed exercise (78 pupils). Of the two questions which formed 
the primary outcome (Exercises 5 and 6), a higher proportion completed Exercise 5 (53 pupils, 42.7%) 
than Exercise 6 (34 pupils, 27.4%). More details relating to the number of pupils completing and 
missing results for each exercise can be found in Appendix 10. 


Primary analysis 


The primary analysis adjusted for baseline-predicted KS2 writing level (Level 3/Level 4). Analysis was 
conducted on 34 pupils (20 intervention pupils and 14 control pupils) who were tested in seven 
secondary schools. There was no statistically significant evidence of a difference in extended writing 
score between the intervention and control groups with a non-significant increase of 0.97 marks for 
those in the intervention group when compared with those in the control group (p = 0.47, 95% Cl: - 
1.76 to 3.71). This relates to an effect size of 0.25 (95% Cl: -0.42 to 0.92). 


Secondary analyses 


Reading 


A regression model was fitted with adjustment for baseline-predicted level only (Level 3/Level 4). 
Analysis was conducted on 40 pupils (24 intervention pupils and 16 control pupils) tested in 12 
secondary schools. There was no evidence of a statistically significant difference in spelling and 
grammar score between the two randomised groups with a non-significant increase of 0.85 marks for 
those in the intervention group when compared with those in the control group (p = 0.51, 95% Cl: - 
1.72 to 3.41). This relates to an effect size of 0.21 (95% Cl: -0.41 to 0.83). 


Spelling and grammar 


A regression model was fitted with adjustment for baseline-predicted level only (Level 3/Level 4). 
Analysis was conducted on 69 pupils (44 intervention pupils and 25 control pupils) tested in 13 
secondary schools. There was no evidence of a statistically significant difference in spelling and 
grammar score between the two randomised groups with a non-significant increase of 0.58 marks for 
those in the intervention group when compared with those in the control group (p = 0.53, 95% Cl: - 
1.26 to 2.42). This relates to an effect size of 0.15 (95% Cl: -0.34 to 0.63). 


Sub-group analyses 


A pre-specified subgroup analysis was planned to assess the impact of the intervention on those 
eligible for FSM. Due to the low completion rate, it was not possible to conduct this analysis. It also 
was not possible to conduct a planned subgroup analysis to investigate the effect of the four teaching 
groups within the intervention arm. 


Post-hoc analyses 


Due to the low response rate, it was not possible to adjust for all the covariates stated in the statistical 
analysis plan. To explore the potential impact of FSM status, EAL status and gender, each was 
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included in a separate repetition of the primary analysis. Results are presented in Table 4. There was 
no evidence of a significant effect of the intervention on extended writing score in any of the analyses 
and neither gender, FSM status nor EAL status had a significant impact on the primary outcomes (p = 
0.37, p = 0.64 and p = 0.57 respectively). 


Table 4. Post-hoc analysis results — primary analysis with further adjustments 


Primary Analysis 0.97 (-1.76 to 3.71) 0.47 0.25 (-0.42 to 0.92) 
Adjusting for 1.20 (-1.48 to 3.89) 0.37 0.31 (-0.36 to 0.98) 
gender 
Adjusting for FSM 0.65 (-2.17 to 3.48) 0.64 0.17 (-0.50 to 0.84) 
status 
Adjusting for EAL 0.83 (-2.09 to 3.75) 0.57 0.22 (-0.46 to 0.88) 
status 


A further post-hoc analysis used total score on Exercise 5 as the response and adjusted for baseline- 
predicted KS2 writing level (Level 3/Level 4), FSM status, gender and EAL status. As the correlation 
between Exercises 5 and 6 was fairly high (0.68) this analysis was deemed appropriate. Analysis was 
conducted on 54 pupils (32 intervention pupils and 21 control pupils) who were tested in 9 secondary 
schools. There was no evidence of a statistically significant difference in total score on Exercise 5 
between the randomised groups with a non-significant increase of 0.29 marks for those in the 
intervention group when compared with those in the control group (p = 0.67, 95% Cl: -1.05 to 1.63). 
This relates to an effect size of 0.13 (95% Cl: -0.41 to 0.68). 


A final post-hoc analysis used the combined total on Exercises 3 and 4 (the short form parts of the 
secondary outcome of reading) as the response and adjusted for baseline-predicted KS2 writing level 
(Level 3/Level 4), FSM status, gender and EAL status. Analysis was conducted on 75 pupils (48 
intervention pupils and 27 control pupils) who were tested in 14 secondary schools. There was no 
evidence of a difference in short form reading score between the randomised groups with a non- 
significant increase of 0.67 marks for those in the intervention group when compared with those in the 
control group (p = 0.46, 95% Cl: -1.13 to 2.47). This relates to an effect size of 0.19 (95% Cl: -0.28 to 
0.66). 


Cost 


Schools were not charged to take part in the project evaluation, however the ongoing cost of the 
programme is estimated at £1,790 per pupil. This estimate includes venue hire, food and travel 
(estimated at £635 per pupil), direct salary costs of staff (£773), promotion and contingency (£94) and 
management and overheads (£288). This estimate is based on 125 pupils attending a summer school 
on a single site. 
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Attendance 


Attendance data were available for pupils allocated to attend the summer school. Over a third of pupils 
did not attend any sessions (28 out of 76, 36.8%) and approximately 10% attended all sessions (7 out 
of 76). The distribution of attendance was non-normal (see Figure 2) and whilst the mode of the 
distribution was 0, nearly half of those allocated to the summer school attended on 15 or more days 
(37/76, 48.6%). 


Figure 2. Attendance 


Histogram of number of days attended 


B 10 15 20 
Number of days attended 


During the first week of the summer school the implementation team attempted to contact the 
parents or carers of the pupils who had not attended to ascertain reasons for non-attendance. This 
was mostly unsuccessful, but in instances where a reason was given this tended to be because their 
child no longer wanted to attend, or because of previously unscheduled holidays, or in some cases 
because the parent was unsure whether they would be attending at a later date. 


Of those pupils who attended some, but not all, sessions of the summer school, a variety of reasons 
were cited, including: pre-booked holidays, hospital appointments, birthday celebrations, induction at 
secondary schools, Eid celebrations, illness and other family commitments such as weddings or 
funerals. 


Process evaluation findings 


Observations were undertaken of two sessions, the first led by a poet and writer with experience in 
delivering creative writing training to children, and the second delivered by two local primary school 
teachers. These sessions were supported by members of the Discover team, termed ‘Story Builders’, 
and by volunteers (mostly trainee teachers wishing to gain experience during their summer break from 
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university). Sessions were 1 hour 15 minutes in length and were held in groups of 14 pupils, although 
this varied daily depending on levels of attendance (minimum 4 pupils per group, maximum 14 pupils 
per group). 


Sessions were pre-planned to meet the broad aims of the Discover Summer School and were based 
around a book upon which much of the summer school teaching was framed. This book, The 
Unforgotten Coat (Cottrell Boyce, 2012), was given to each child at the summer school and described 
the experiences of a central character in their final year at primary school and period before starting 
secondary school. This appeared to engage the children through its use of imaginative imagery and 
photography, and was of relevance to the ‘transition’ period that the pupils were experiencing. The 
book also discussed issues such as immigration and minority groups that may have been pertinent to 
this group. 


In poet-led literacy sessions, techniques were used to engage the children as much as possible, for 
example using actors and objects in the classroom to excite their imagination and encourage creative 
writing. Pupils were then asked to explore these objects together as a group and write about them, 
giving brief histories and descriptions of each using an imaginative approach. In another session 
pupils were asked to research a famous poet or writer using online searching and then write 
descriptions about these people. Children were given advice about thinking creatively and expanding 
their descriptions to aid their writing. Children were encouraged positively, for example: ‘You can’t get 
this wrong, don’t worry, it’s your ideas.’ 


Children for the most part appeared well engaged and to enjoy these sessions. The interactive 
approach in poetry sessions in particular appeared to motivate pupils in their writing. Children talked 
and listened to each other as a group at the beginning and end of sessions, allowing them to build 
confidence in sharing their ideas to a group. Attempts were made to involve all pupils, particularly the 
shy pupils in the group. At times there were pupils who were not engaged in the class, and this was 
more apparent in one group than in another, possibly as, by chance, these children had been grouped 
together. 
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Conclusion 


Conclusions and implications 


The conclusions that can be drawn from this study are limited due to the small sample size and 
problems with attrition. We found no evidence that the Discover Summer School had a statistically 
significant effect on English attainment. There is, however, a need for further research to explore the 
effect of literacy summer schools such as this, and in this respect key lessons from this study may 
prove useful. 


Strengths of the evaluation 


In the design and conduct of our study we used best practice as defined by the CONSORT guidelines 
for randomised controlled trials (Altman et al., 2011). Importantly, we used independent concealed 
allocation to ensure that the children were allocated without the possibility of bias. We used the 
principles of intention-to-treat by including all consenting eligible children in the final analysis. We pre- 
specified our main outcome and wrote a statistical analysis plan before we observed the data. We also 
used an independent company to mark the test papers, blind to the allocated group. 


Limitations of the evaluation 


The target sample size of 250 pupils was not achieved in this study, with only 124 eligible pupils 
recruited into the study and attrition from testing in this group further reduced our ability to detect an 
effect of the summer school intervention, as statistically significant, if one existed. 


Potential explanations for this low recruitment appear to be twofold. Firstly, it is possible that the 
design of the intervention may not have attracted parents to consent for their child to be part of the 
study, for example as the summer school took place five days per week over the course of four weeks 
during the summer holidays. Day et al. (2013) suggest that while literacy summer schools need to be 
sufficient in duration to improve pupils’ literacy skills, summer schools over two weeks in length may 
not be feasible due to clashes with family holidays and other commitments, as well as reducing levels 
of interest amongst pupils. Another potential explanation for low recruitment to this trial is the lateness 
with which parents were informed of the study. Owing to the short timescale for this trial, recruitment 
took place in late spring, when many may have already made plans for the summer holidays and so 
could not commit to the summer school intervention. Future summer schools and evaluations may 
wish to consider a shorter intervention and greater advance notice for parents considering their child’s 
participation. 


Attrition was also a key problem in this evaluation: this occurred both in terms of non-attendance at the 
summer school and in terms of non-completion of the test. We explored the impact of non-test 
completion in a post-hoc analysis comparing the mean values (Table 4) of the GL scores that had 
been completed by the highest proportion of children. However, this did not change the overall finding 
of a lack of statistical evidence for a difference in post-test scores. This study has shown that it is not 
advisable to undertake a full trial of this intervention until issues associated with attrition from the 
intervention are resolved. 


Further work needs to be done to investigate approaches that may overcome problems with low 
attendance, as 36.8% of the intervention group pupils never attended. This was despite steps that 
were taken by the implementation team to enable greater attendance, for example through the 
provision of free transport arrangements for pupils from multiple collection and drop-off points. Day et 
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al. (2013) recently reported similar problems with attendance (50% of disadvantaged pupils never 
attending) suggesting that this is a common difficulty in undertaking summer schools with children 
from disadvantaged groups. As aforementioned, a different format of summer school may be required 
as it is possible that parents or guardians may have been unwilling or unable to commit to their child to 
attend a four-week summer school. 


In terms of attrition from testing, primary outcome data for fewer than 30% of trial pupils was available. 
To overcome this problem, we recommend that researchers should explore the use of whole-class 
testing or using routinely collected data for outcome assessment, such as the standard assessment 
tests (SATs) where data collection is planned to be undertaken by schools and that this is thought to 
pose a particular problem. This recommendation may be useful as lessons learned from this 
evaluation (where schools may have experienced difficulty in testing only a handful of students in 
some instances and finding adequate time for children to sit the tests, resulting in incomplete tests and 
missing data) could prevent similar problems occurring in future trials. 


Attrition from testing in this evaluation has also highlighted the important role that schools have in 
administering tests and emphasis should be placed on sufficient engagement with schools to ensure 
that their role is fully understood. This may be particularly necessary in trials that explore interventions 
taking place during the transition from primary to secondary school, where initial priority may be 
placed on liaison with primary schools to identify pupils and supply baseline data, but sufficient 
emphasis also needs to be placed on the involvement of secondary schools undertaking outcome 
assessment from the outset, so that both sets of schools are sufficiently engaged and aware of the 
requirements for their involvement. Transitions projects in their very nature also make the process of 
identifying schools and pupils difficult, with changes to eligible pupils’ secondary schools, in this trial 
resulting in 62 consenting pupils being excluded from the evaluation. 


Although this study concentrated on investigating the effect of a literacy summer school on children 
predicted to achieve Level 3c to Level 4c in English by the end of Key Stage 2 (based on teacher 
assessment), upon analysis of baseline characteristics it was noted that three pupils were included in 
error (one at Level 4b and two at Level 4a), however as the intervention was not inappropriate for 
these pupils they were retained in the analyses. 


Generalisability of results 


The generalisability of this study is limited by the relatively small geographic location in which the 
schools were based. Nonetheless, it is likely that the pupils sampled may be representative of other 
inner-city urban areas which have a high proportion of pupils belonging to minority ethnic groups or 
eligible for FSM. It is possible that attendance rates at this summer school might differ from areas in 
the UK with lower proportions of disadvantaged or ethnic minority groups, for example children that 
were fasting for Ramadan during the summer school may have struggled with their attendance. This 
should be borne in mind when planning future summer schools. Low attendance rates at the summer 
school may limit the generalisability of these findings to those children that are more likely to attend a 
summer school; for example the sample in the evaluation may represent children and parents with 
greater motivation and may not include harder to reach children and parents. 


Further research 


The need for further research exploring the feasibility of conducting a larger RCT in this field has been 
highlighted. This work should also consider the potential effect that other activities over the summer 
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holidays may have on children’s attainment, and could be investigated through questionnaire methods 
and incorporated into statistical analyses. 


Although a process evaluation was conducted in this study to observe the intervention being delivered, 
this formed a very small part of this evaluation and further qualitative research may be valuable to 
identify other aspects of pupils’ progress that could benefit from participation in a summer school. 
Domains of interest may include pupils’ confidence in their literacy, general self-esteem or attitudes 
towards the transition from primary to secondary school. Additionally, qualitative interviews or focus 
groups with pupils, parents and teachers may be of use to gain insight into the potential barriers and 
facilitators of conducting future evaluations of summer schools. 


Given the large number of projects currently being funded by bodies such as the EEF, trialists 
undertaking educational research should be aware of the potential for pupils to be participating across 
multiple studies. During this evaluation it was noted by the trial statistician, working on multiple 
educational trials, that two participating pupils had also been included in another ‘transitions trial.’ As 
the intervention and testing periods for both trials did not overlap, the decision was taken to retain 
these participants. However, as the same test was used in both trials, these pupils may have benefited 
from ‘training’ in the test and future trials should consider exploring the effect of this through sensitivity 
analyses. Due to the small amount of data in the present trial this was not undertaken. 


The full involvement of the independent evaluators in the recruitment of pupils and schools is 
recommended for future trials in order to ensure that the nature of the evaluation, the randomisation 
process and the importance of outcome assessment is sufficiently discussed in order to circumvent 
potential problems such as attrition from testing that was experienced in this trial. For example, some 
teachers thought that children who did not attend the summer school did not need to be tested. An 
independent evaluator who had recruited the schools and teachers would have made this clearer and 
consequently may have avoided this problem. This would also enable greater monitoring of 
anticipated recruitment rates so that additional measures could be taken in order to reach the desired 
sample size. 


Conclusion 


This randomised controlled trial explored the effect of a literacy-based summer school on the reading 
and writing abilities of pupils. Although the conclusions that can be drawn from this study are limited 
due to the small sample size and problems with attrition, the feasibility of undertaking such a study on 
a larger scale has been explored and useful points for consideration prior to undertaking a full trial are 
recommended. 
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Appendices 


Roles and Responsibilities 


Evaluation Team 


Chief Investigators: Professor David Torgerson (DT), Director, York Trials Unit, Department of 
Health Sciences, University of York, Heslington, York, YO10 5DD. T: 01904 


321340 E: david.torgerson@york.ac.uk 


Professor Carole Torgerson (CT), School of Education, Durham University, 


Leazes Road, Durham, DH1 1TA. T: 0191 334 8382 E: 
carole.torgerson@durham.ac.uk 
Co-Investigator: Hannah Ainsworth (HA), Research Fellow, York Trials Unit, Department of 


Health Sciences, University of York, Heslington, York, YO10 SDD. T: 01904 
328158 E: hannah.ainsworth@york.ac.uk 


Trial Manager: Laura Jefferson (LJ), Research Fellow, York Trials Unit, Department of Health 
Sciences, University of York, Heslington, York, YO10 5DD. T: 01904 321511 E: 


laura.jefferson@york.ac.uk 


Trial Statistician: Hannah Buckley (HB), Trainee Statistician, York Trials Unit, Department of 
Health Sciences, University of York, Heslington, York, YO10 SDD. T: 01904 


321512 E: hannah.buckley@york.ac.uk 


Researchers: Natasha Mitchell (NM), Research Fellow, York Trials Unit, Department of 
Health Sciences, University of York, Heslington, York, YO10 5DD. T: 01904 


321655 E: natasha.mitchell@york.ac.uk 


Clare Heaps (CH), Research Associate, School of Education, Durham 
University, Leazes Road, Durham, DH1 1TA. T: 0191 384 8332 E: 


C.N.Heaps@dur.ac.uk 


The evaluation team will be responsible for the design, conduct, analysis and reporting of the 
independent evaluation. 


DT and CT — Design of trial; write protocol; oversee all stages in the design, conduct, analysis and 
reporting of trial, including recruitment and retention of schools, report-writing; supervise work of 
trial manager and researchers, statistician and data managers on the trial. 


LJ — Design of trial; write protocol; register trial; trial co-ordination and data management, 
contribution to the analysis and write up. 


HA and NM -Trial co-ordination assistance. 


HB — Design of trial, write trial analysis plan, randomisation, conduct analyses, contribute to write up. 
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CH — Assist with trial co-ordination and data management; co-ordinate independent outcome 
assessments 


Sponsor 


Sue Final, Intellectual Property Manager, University of York, Research Innovation Office, Innovation 
Centre, York Science Park, York, YO10 5DG. T: 01904 435154 F: 01904 435101 E: 
sue.final@york.ac.uk 


Implementation Team 


Discover Children’s Story Centre. Registered Charity No: 1070468 
Address: Discover Children's Story Centre, 383-387 High Street, Stratford, London, E15 4QZ. 


Contacts: 

Pippa Parsons 
Project Manager 
Discover 

383-387 High Street 
Stratford 

LONDON 

E15 4QZ 


T: 020 8536 5554 
E: pippa.parsons@doscver.org.uk 


Dawn Ingleson 
Learning Manager 
Discover 

383-387 High Street 
Stratford 

LONDON 

E15 4QZ 


T: 020 8536 5546 
E: dawn.ingleson@discover.org.uk 


Discover Children’s Story Centre (implementation team) will be responsible for school recruitment 
and on-going relationship with schools, informing parents and pupils, intervention development, 
including writing detailed description of intervention to allow others, if necessary, to be able to 
replicate the intervention in other areas, intervention training and delivery, baseline data collection. 


Funder 
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Education Endowment Foundation (EEF), Registered charity 1142111, Millbank Tower, 21-24 
Millbank, London SW1P 4QP. 


Emily Yeomans, Grants Manager, EEF. T 020 7802 0644 E: emily.yeomans@eefoundation.org.uk 
Camilla Neville, Evaluation Manager, EEF. T: 020 7802 0640 E: Camilla.Nevill@eefoundation.org.uk 


The Education Endowment Foundation has funded both the Transitions Project (Improving Writing 
Quality) and the independent evaluation. They will have oversight of the project. 


Ethics Committee 


Durham University School of Education Ethics Committee 


York Health Sciences Research Governance Committee (by Chair’s Action) 


Advisory Group 


Durham University School of Education and CEM EEF panel of evaluators and York Trial Unit panel of 
evaluators will provide advice on any aspect of the design, conduct and reporting of the evaluation 
when and as required. 


Background and significance 


The Education Endowment Foundation has funded the University of York and Durham University to 
evaluate the Discover Children’s Story Centre summer writing workshop intervention being delivered 
and supported by Discover Children’s Story Centre. 


The intervention will provide summer writing workshops to children between Year 6 and the start of 
Year 7. The intervention will involve a four week summer programme for children in order to 
improve reading and writing skills, specifically of children who are less able writers. The Programme 
will include a variety of workshops, including poetry and literacy sessions, in which children will also 
have the opportunity to engage with professional poets and writers. A range of enrichment activities 
will also be provided including: drama, sports activities and trips around London, such as West End 
performances and visits to the Olympic Park. An independent evaluation using rigorous design and 
methods is timely as there have been no previous evaluations of the impact of this project. 


The trial will compare the summer writing workshop intervention with a “business as usual” control 
group who will not receive the summer writing workshop. Such a comparison will allow us to 
estimate whether or not there is an important intervention effect. We propose to recruit 250 pupils 
for this evaluation. 


Education Endowment Foundation 29 


7 Vo) of-ale|[eq--) 


The implementation team (Discover) will be responsible for developing and delivering the summer 
writing workshop intervention. They will provide a detailed description of the intervention and a 
manual to be used as guidance for these summer schools. 


Research Question 


What is the effectiveness of the Discover summer writing workshop intervention compared with a 


|’ 


“business as usual” control group on the reading and writing abilities of participating children? 


Design 


This will be a pragmatic individually randomised controlled trial involving 250 children. We estimate 
that there will be approximately 10-12 children, per primary school class of approximately 27 
children, who will be potentially eligible for this study. The average primary school size in the 
geographical area will determine the number of primary schools that will be required to achieve this 
sample size. 


Pupils in Year 6 will be eligible to be included in the trial if they 1) attend a participating secondary 
school and 2) are predicted to achieve between level 3c and level 4c in English by the end of Key 
Stage 2 (based on teacher assessment). 


Pupils whose parents provide informed consent for them to participate in the trial will be 
randomised to either the intervention or control. The intervention group will be invited to attend a 
Discover Children’s Story Centre four week summer programme, which will take place at one hosting 
secondary school. Pupils attending the summer school will then be randomly allocated into one of 
five teaching groups. The control group will not attend the Discover Children’s Story Centre four 
week summer programme, however in order to increase recruitment and avoid disappointment the 
control group will receive a secondary school-based reading and writing intervention delivered in 
the autumn term 2013. This secondary school-based intervention will not form part of the 
evaluation as it will take place after the outcome testing. 


The trial will be designed, conducted and reported to CONSORT standards (Altman et al, 2011) in 

order to minimise all potential threats to internal validity, such as selection bias and a range of post 
randomisation biases (Cook and Campbell, 1969; Shadish, Cook and Campbell, 2002; Torgerson and 
Torgerson, 2008). In this way, unbiased estimates of the impact of the intervention will be provided. 


Recruitment 
Discover will recruit appropriate schools and pupils. Recruitment will take place during the 


development phase of the intervention (i.e. between March and May 2013). The York/Durham 
evaluation team will prepare information documentation about the evaluation for schools and 
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pupils/parents. This will include details of the trial design. The intervention will be delivered during 
the summer break between Year 6 and Year 7. 


Inclusion criteria 


Participating secondary schools will liaise with primary schools to provide a list of pupils that will be 
attending in the autumn term 2013. Primary schools will screen this list of pupils to identify potential 
pupils meeting the following inclusion criteria: 


1) Attend a participating secondary school 
2) Predicted to achieve between level 3c and level 4c in English by the end of Key Stage 2 
(based on teacher assessment). 


Primary schools will inform the parents of eligible pupils about the study (material and help provided 
by the evaluation team and Discover charity). Parents will then be asked to provide written informed 
consent to agree for their child to participate in the study. 


School participation 


In order for schools to be eligible to take part in the evaluation and to receive the intervention, we 
will put in place a memorandum of understanding with the schools which will specify the following: 


Primary schools: 

e Enthusiasm for the project and for professional development 

e Agreement to be in the independent evaluation 

e Willingness to allow the random allocation of Year 6 pupils to either intervention or comparison 
group 

e Willingness to identify all eligible pupils using pre-specified criteria 

e Provision of baseline data about pupils 


Secondary schools: 

e Enthusiasm for the project and for professional development 

e Agreement to be in the independent evaluation 

e Liaison with feeder primary schools to identify pupils that will attend in Year 7 

e Willingness to administer the outcome assessment to children participating in the trial in Year 7 
within 2 weeks of the summer intervention ending (the test will be marked independently by 
external markers) 


Randomisation 


An independent individual at York Trials Unit will use a dedicated computer program to carry out all 
randomisation. Once parents and pupils have given consent their details will be sent to the York 
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Trials Unit and these will be entered onto a randomisation database. Pupils will be individually 
randomised to either the intervention or control group. If recruitment is as predicted, this 
randomisation will use equal allocation. Due to there being a limited number of places on the 
summer school, if recruitment is better than expected, unequal allocation will be used with 125 
pupils allocated to the intervention group and with those remaining allocated to the control group. 
If recruitment is not as good as anticipated, unequal allocation will favour the intervention group in 
order to fill all the places on the summer school. 


Pupils randomised to the intervention group will be randomised further into five groups for teaching 
purposes using equal allocation. Pupils randomised to the control group will be pseudo randomised 
to one of the five teaching groups for analysis purposes using equal allocation. 


Sample size calculation 


This trial will aim to recruit 250 pupils. To estimate the differences that we could estimate, with 80% 
power (2p = 0.05), we use the assumption that there will be a pre and post-test correlation of 0.70 
(i.e. the outcome test will correlate with the Key Stage 2 English result with a correlation of 0.70). 
This means that we have an effective sample size of 510 pupils. Allowing for a 10% attrition rate give 
an effective sample size to 460 pupils. This would give 80% power to show a difference of 0.27 
standard deviations in writing score between the intervention and control groups should a difference 
of this magnitude exist. 


Analysis 


Analysis will be conducted using the principles of intention to treat, meaning that all pupils will be 
analysed in the group to which they were randomised irrespective of whether or not they actually 
received the intervention. 


Statistical significance will be assessed at the 5% level unless otherwise stated and 95% confidence 
intervals will be provided as appropriate. Methods for handling missing data and further detail on 
analyses will be provided within a statistical analysis plan. 


Primary Analysis 


The primary objective of this study is to investigate the effectiveness of the intervention on the 
writing skills of eligible children. The difference between pupils receiving the Discover four week 
summer programme intervention and those pupils in the control group will be compared using 
regression analysis with writing post-test result as the response variable. Group allocation, predicted 
KS2 score and gender will be included as covariates in the model. 


Secondary Analyses 
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A similar method to that taken in the primary analysis will be used to assess differences between the 
intervention and control groups in terms of the secondary outcomes of reading, spelling and 
grammar. 


Subgroup analyses will investigate the effect of the intervention on pupils who are eligible for FSM in 
terms of post-test scores. 


The impact of non-compliance (should this occur) will also be assessed, using Complier Average 
Causal Effect (CACE) analysis, which will estimate the intervention’s impact on writing skills in the 
absence of non-compliance. 


An ancillary analysis will compare difference in terms of post-test scores between the intervention 
and control pupils assigned to the five teaching groups using a paired t test. 


Outcome measure 


We propose to use the Progress in English 11 (Long Form), GL Assessment, for our main literacy 
outcome. The primary outcome will be the combined score on the two writing tasks within the test. 
Secondary outcomes will be scores on the reading, spelling and grammar components of the test. 
The assessment will be carried out within 2 weeks of the intervention ending, in local secondary 
schools. All tests will be completed during the first three weeks of the autumn term 2013. Long term 
outcomes can be collected through the National Pupil Database. 


Test results will be marked, blind to group assignment, by GL assessment. The results will then be 
sent to Durham University and copies of the results will be forwarded to the York Trials Unit for 
analysis. 


Process evaluation 


Independent observer(s) will attend at least one session of the Discover Children’s Story Centre 
summer workshop to observe the delivery of the intervention. 


Data Protection Statement 


Durham University’s data protection policy is publically available at: 
http://www.dur.ac.uk/resources/data.protection/dataprotectionpolicy.pdf 


“Durham University is committed to protecting the rights and freedoms of individuals in accordance 
with the provisions of the Data Protection Act 1998. The requirements to which University staff and 
student who process personal data must adhere are set out in the University’s Data Protection 
Policy” 


The University of York’s data protection policy is publically available at: 


Education Endowment Foundation 33 


Appendices 


http://www.york.ac.uk/media/recordsmanagement/documents/dataprotectiondocs/Data%20Protec 
tion%20Policy.pdf 


Risks 


Low risks associated with this project include operational and project specific risks. For the 
Operational risks such as staffing and IT / assessment system we are confident that we have systems 
and procedures in place to minimise any risks, but would nonetheless be very happy to provide 
further details. 


School and pupil recruitment — whilst this will be the primary responsibility of the implementation 
team, the evaluation team have a good track of recruiting schools, and will help with this if 
necessary. 


Attrition and loss to follow up — in a study such as this it is essential that this is kept to a minimum. 
Whilst this will be the primary responsibility of the implementation team, our involvement, as with 
the recruitment, will help to minimise attrition. 


Maintaining fidelity (intervention and control) - it is essential that as many as possible schools 
maintain a high level of implementation fidelity. Again whilst the primary responsibility will lie with 
the implementation team we will provide some support, for example, by emphasising the 
importance of contributing to the process of building good evidence. 
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Independent Evaluation of the Discover Children’s Story Centre summer 
writing workshop 
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1. Trial Objectives 


Primary objective 


The primary objective of the trial is to evaluate the effectiveness of the Discover summer writing 
workshop intervention compared with a “business as usual” control group on the reading and 


writing abilities of participating children. 
2. Design 


The study is a pragmatic, two-armed randomised controlled trial, with randomisation at the pupil 
level. The intervention group will be invited to attend a Discover Children’s Story Centre four week 
summer programme. The control group will not receive any reading or writing intervention until 
after the outcome testing. 

Pupils in Year 6 will be eligible to be included in the trial if they 1) attend a participating secondary 
school and 2) are predicted to achieve between level 3c and level 4c in English by the end of Key 
Stage 2 (based on teacher assessment). 

Pupils attending the summer school will be taught in one of five teaching groups. 

The trial will be designed, conducted and reported to CONSORT standards (Altman et al, 2011) in 
order to minimise all potential threats to internal validity, such as selection bias and a range of post 
randomisation biases (Cook and Campbell, 1969; Shadish, Cook and Campbell, 2002; Torgerson and 


Torgerson, 2008). In this way, unbiased estimates of the impact of the intervention will be provided. 


3. Sample Size 


The trial will aim to recruit 250 pupils. To estimate the difference that we could detect, with 80% 
power (2p = 0.05), we use the assumption that there will be a pre and post-test correlation of 0.70 
(i.e. the outcome test will correlate with the Key Stage 2 English result with a correlation of 0.70). 
This means that we have an effective sample size of 492 pupils. Allowing for 10% attrition reduces 
the effective sample size to 442 pupils. This would give 80% power to show a difference of 
approximately 0.27 standard deviations in writing score between the intervention and control groups 


should a difference of this magnitude exist. 


(NB. There was a slight error in the minimal detectable difference calculation in the published 


protocol. This does not affect the detectable effect size and the correct version is printed here.] 
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4. Randomisation 


Pupils will be individually randomised to either the intervention or control group. If recruitment is as 
predicted, this randomisation will use equal allocation. Due to the a limited number of places on the 
summer school, if recruitment is better than expected, unequal allocation will be used with 125 
pupils allocated to the intervention group and with those remaining allocated to the control group. 
If recruitment is not as good as anticipated, unequal allocation will favour the intervention group in 


order to fill all the places on the summer school. 


Due to the trial under-recruiting, randomisation will be conducted using a 3:2 (Intervention:Control) 
allocation ratio within each secondary school. A fixed block size of 10 will be used. There is no 


concern about predictability as all pupils will be randomised at one time. 


There was a request that twins would have the same allocation. Both pupils will be randomised 
separately and then both will receive the allocation assigned to the ‘first’ twin as defined by the 


smallest trial ID. 


It was planned that pupils allocated to the intervention would be taught in 5 colour groups and that 
they would be assigned to these groups randomly on a 1:1:1:1:1 ratio. This did not occur due to 
Discover’s preference, this was not thought to be a problem as we are examining the effectiveness of 


the intervention in practice. Also, teaching was conducted in 4 groups and not 5. 


5. Definition of terms 
Level 3 or fragile Level 4: Level 3c, Level 3b, Level 3a, Level 4c or Level 4b 
KS2: Key Stage 2 
(KS2) SAT: (Key Stage 2) Standard Assessment Tests 


PiE 11 (LF): Progress in English 11: Second Edition (Long Form) sometimes simply referred to as PiE or 


PiE 11 
Raw score: number of marks awarded (no adjustment for age or similar) 


Extended writing score: the combined raw score on the two extended writing tasks on the PiE11 
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Overall writing score: the combined raw score on the spelling, grammar and writing tasks on the 


PiE11 

National curriculum writing level: Key Stage 2 SAT Writing Level 

Reading score: the combined raw score on the reading tasks on the PiE11 
National curriculum reading level: Key Stage 2 SAT Reading Level 


Spelling and grammar score: the combined raw score on the spelling and grammar tasks on the 


PiE11 

Overall raw score: combined total raw score on the PiE11 

Standardised age score: overall raw score on the PiE11 adjusted for age in months 
FSM: Free School Meals 

PP: Pupil premium 


EAL: English as an Additional Language 


6. Outcomes 


The Progress in English (PiE) 11: Second Edition (Long Form), GL Assessment, will be the main literacy 
outcome. The test includes both narrative and non-narrative exercises and assesses both reading and 
writing skills including areas such as spelling, grammar and comprehension. The Progress in English 
test was the only test available to the evaluation team (in order to comply with EEF testing policy) 


which included a writing component. 


Primary outcome 


The primary outcome will be the combined raw score on the two extended writing tasks (Ex 5 and Ex 


6). This test will be conducted within 2 weeks of the intervention ending. 


Exercise 5 has a total possible 20 marks. These marks come from 4 sub scores: 


e 5.1 Composition and effect (8 marks) 


e 5.2 Text organisation (4 marks) 
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e 5.3 Sentence structure (4 marks) 
e 5.4 vocabulary (4 marks) 


Exercise 6 has a total possible 12 marks. These come from 2 sub scores: 


e 6.1 Composition and effect and vocabulary (8 marks) 
e 6.2 Text organisation and sentence structure (4 marks) 
The combined raw score on the two extended writing tasks is in the range O to 32, with a higher 


score representing better attainment. 


Secondary outcomes 


Reading 


The combined raw score on the reading tasks (Ex 3, Ex 4, Ex 3x and Ex 4x) will be a secondary 


outcome. 


Exercise 3 (comprising of Ex3 and Ex 3x) has a total possible 19 marks and assesses reading 
comprehension of a narrative. Exercise 4 (comprising of Ex4 and Ex 4x) has a total possible 13 marks 
and assesses non-narrative reading comprehension. Therefore, the combined raw score on the 


reading should be in the range O to 32, with a higher score representing better attainment. 


The corresponding national curriculum writing level can be obtained as follows: 


Writing Raw Score Level 
0-11 3 or below 
12-15 4c 
16-18 4b 
19-21 4a 
22-23 5c 
24-25 5b 
26-32 5a or higher 


Spelling and Grammar 


The combined raw score on the spelling and grammar tasks (Ex 1 and Ex 2) will be a secondary 


outcome. 
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Exercise 1 has a total possible 10 marks and assesses spelling. Exercise 2 has a total possible 10 marks 
and assesses grammar. The combined raw score on the reading is in the range O to 20, with a higher 


score representing better attainment. 


Other important information 
Important pupil characteristics include: 


e Gender 

e FSM status 
e PP status 

e EAL status 


e Predicted KS2 Writing Score 


7. Analysis 


Analysis will be conducted using the principles of intention to treat, that is, all pupils will be analysed 
in the group to which they were randomised irrespective of whether or not they actually received 
the intervention. Statistical significance will be assessed at the 5% level unless otherwise stated and 
95% confidence intervals will be provided as appropriate. Analyses will be conducted on complete 


cases. 


Baseline data 


All participant baseline data will be summarised descriptively by trial arm. No formal statistical 
comparisons will be undertaken. Continuous measures will be reported as means and standard 


deviations, while categorical data will be reported as counts and percentages. 


Primary analysis 


Regression analysis of the post-test extended writing score on randomised group will be carried out. 
Gender, FSM status, EAL, age and predicted KS2 score will be included as covariates in the model. 
Adjustment will be made for secondary school which was used as a stratification factor. The estimate 


of the difference in scores between pupils randomised to the Discover four week summer 
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programme and pupils randomised to the control group will be reported, together with the 95% 


confidence interval. 


Analysis of the primary outcome will be checked by a second statistician. 


Secondary analyses 


Regression analysis adjusting for the same covariates as in the primary analysis will be used to assess 
differences between the intervention and control groups in terms of the secondary outcomes of 


reading, spelling and grammar. 


Subgroup analyses 


A subgroup analysis will investigate the effect of the intervention on post-test scores for pupils who 
are eligible for FSM. An interaction term between FSM status and randomised allocation will be 
included in the regression model fitted in the primary analysis. Statistical significance will be set at 


the 10% level as this trial is not powered to detected interactions. 


A subgroup analysis will compare difference in post-test scores between the intervention and control 
pupils assigned to the four teaching groups. An interaction term between teaching group and 


randomised allocation will be included in the regression model fitted in the primary analysis. 


Sensitivity analyses 


Patterns of missingness in the post-test result data will be explored. Where appropriate multiple 
imputation will be considered provided the assumption that data are missing at random (MAR) is 


plausible. 


Summary statistics for the number of sessions attended will be produced. 


The impact of non-compliance (should this occur) will also be assessed, using Complier Average 
Causal Effect (CACE) analysis, which will estimate the impact of the intervention among the group of 


compliers. 
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Briefing Sheet for Schools 


Evaluation of the Discover Literacy Summer 
School intervention 


The Education Endowment Foundation (EEF) have 
asked researchers at the University of York and 
Durham University to evaluate a Literacy Summer 
School intervention being delivered and supported by 
the Discover Children’s Story Centre in 2013. The 
summer school intervention will take place during a 
four-week period and will combine literacy sessions 
with enrichment activities (including a range of exciting 
experiences such as trips to West End performances, 
opportunities to meet with professional writers and 
poets and sporting activities). Everything, including 
healthy meals, will be provided free of charge. Lister 
Community School will host the intervention, which will 
be delivered by the Discover team from 29" July to 23 
August 2013. 


The evaluation aims to find out if the intervention 
improves pupils’ reading and writing skills during the 
transition from Year 6 to Year 7, especially the writing 
skills of pupils who are currently achieving between 
KS2 Level 3c and 4c in English. In order to find out how 
well the Literacy Summer Schoo! works, we must 
compare children attending the summer school to 
those that do not. This has to be decided completely at 
random using a computer, and those children who are 
not allocated to receive the intervention will be offered 
a school-based Saturday morning literacy workshop in 
the autumn term of Year 7. 


We aim to recruit 250 pupils for this evaluation, 125 of 
whom will receive the intervention and 125 of whom will 
form the control group. All primary schools who decide 
to take part in the intervention and its evaluation will be 
asked to provide information to the University of York 
about all pupils that consent to take part in the study. 
The information will include each pupil’s UPN and their 
predicted KS2 teacher assessment in English plus 
some demographic information (e.g. gender, date of 
birth, Free School Meal status). Some basic 
demographic information about the school will also be 
requested at this stage. 


At the beginning of autumn term 2013, all 250 
participating Year 7 pupils in participating secondary 
schools will be tested so that researchers at the 
University of York can estimate the effect the 
intervention has had on pupils’ writing skills by 
comparing results from children who attended the 
summer school to those who did not. The GL 
Assessment Progress in English (PiE) 11 (Long Form) 
will be used. 
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What commitment would this project require from 
schools? 


e Enthusiasm for the project 

e Willingness to provide feeder primary schools with 
information about eligible pupils attending in Year 7 
(secondary schools) 

e Willingness to identify, provide information letters 
and collect consent forms for eligible children in 
Year 6 (primary schools) 

e Provision of baseline data about pupils in Year 6 
(primary schools) 

e Willingness to administer a writing test to all 
participating Year 7 pupils during the first weeks of 
autumn term 2013 (the test will be marked 
independently by external markers) (secondary 
schools) 


As a designated EEF Research Partner School you will 
be entitled to: permission to use the EEF’s logo on your 
website, a certificate showing that the school is an EEF 
Research Partner School, access to EEF networking 
and knowledge events, a regular newsletter updating 
schools on the latest EEF projects and evidence, free 
places for Year 6 teachers on a literacy INSET to be 
held at Discover in the autumn term. 


When will this project take place? 


We hope to inform parents of potential pupils in May 
and collect consent for their children to take part in this 
evaluation around May half term. For all consenting 
pupils, baseline data will be collected and pupils 
randomised in June. Parents will be notified of their 
child’s allocation to intervention or control group in July 
and the intervention will take place between 29" July 
and 23” August. Participating pupils will be tested in 
the first 2-3 weeks of the autumn term 2013, after 
which point pupils in the control group will be offered a 
Saturday morning literacy session. 


Is there funding to support my involvement? 


Yes — the Discover Children’s Centre will provide all 
activities, travel and meals free of charge. The 
Saturday morning literacy sessions delivered to control 
group children in the autumn term will also be provided 
free of charge. 
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Appendix 4: Primary School Agreement to Participate Form 


AB Education e Children’s Story Centre 
THE UNIVERSITY Of York ae - -rsownent PIG COV ver 


Foundation 


Evaluation of the Discover Literacy Summer School intervention 
Primary School Agreement to Participate 


| confirm that | have read and understood the information sheet for the above evaluation 
and have had the opportunity to ask questions; 


| understand that all children’s results will be kept confidential and that no material which 
could identify individual children or the school will be used in any reports of this evaluation; 


| agree to identify eligible pupils in Year 6; 
| agree to providing an information letter to all parents of eligible children in Year 6; 


| agree to collect in parent consent forms; 
| agree to provide baseline data about participating pupils in Year 6; 


| understand that pupils’ test responses will be collected by GL Assessment and accessed by 
the evaluation team. Named data will be matched with the National Pupil Database and 
shared between the evaluation team, Discover Children’s Story Centre and EEF. 


| consent to the school taking part in the above study 


PILI WW 


Natmiesar NEARteaC NER veicuicacngeniiatiiareninids Geen mnaielaseadnuceusniuln a anueunanien 
PAF OE SIMO I asscacuin has icetnnesiseteninrmrcngh tac tabaaaie de aunts icborenselanhdahipoe nes glen agaapenasncbinolaangonnctiammanindehes 
SEM Ol Ve WO) a sipicamsts yaricrnnty dae ncemananumi eins wurtieenlannaiis wise ieinsnanuiaradanseueneundne 
Headteacher Ertiall AGAKeSS siijsiascusasiiispacasasaauvanstinncancehagnss toansavanasay ane diaabiceasea uccanoupniaawansmnmnani@aaianns 
Name of School Contact (if Not headteacher)... csssssesseecsssssesssesceecsecessesaesseseeecaeseseesseeceecaeseeeesas 
SMOG CCATACE SAV iAN) ACU SS ices suteauesninss eaigonesccevandhtsnnnldinennunaiinrs ela daccia nsaiecd sun ae cepanaiondivciigencsupouhtexavanianer 
Signature Of headteache’...........ccssscsscsssessecsscssscsscsssecsscessesseeceeseeacseesseesseeees Dats svcinedisensssccesaciesses 


Thank you for agreeing to take part in this research. Please return this consent form by post 
to: Mrs D Parsons, Discover Children’s Story Centre, 383-387 High Street, Stratford, London, 
E15 4QZ 
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Appendix 5: Secondary School Agreement to Participate 
Form 


Education e Children’s Story Centre 


ae A 
D ham ndowmen 
THE UNIVERSITY Of York Ba ac -vsowren' | IG COV ver 


Evaluation of the Discover Literacy Summer School intervention 


Secondary School Agreement to Participate 


| confirm that | have read and understood the information sheet for the above evaluation 
and have had the opportunity to ask questions; 


| understand that all children’s results will be kept confidential and that no material which 
could identify individual children or the school will be used in any reports of this evaluation; 


attending the school in Year 7 in the autumn term on 2013; 


| agree to administer a writing test to all participating Year 7 pupils in September 2013 


| understand that pupils’ test responses will be collected by GL Assessment and accessed by 
the evaluation team. Named data will be matched with the National Pupil Database and 
shared between the evaluation team, Discover Children’s Story Centre and EEF. 


[| 
[| 
| | | agree to provide feeder primary schools with information about the pupils that will be 
[| 
[| 
| | | consent to the school taking part in the above study 

Namesof Neate ache wisiccsscisscvstschsasescsaivenseassieitusseesdabi ies dss chesicatbosctlathe essdeevivsneasasiacieaastaccaedivntuvacuaedivnis 
Name Of SCHOON ssies si ecetes cnet cSel dds saccecguekig sivessctedda Shu goandg siegdidadecenducd dasdscutet cipatesnagdbenteudiidetesscsagueaines 
SCHOO! Tel. NO sssssteesevssestexeicsevevacesdiatecseecenstesvebesddtaashi ons sdessecedasdsiastscavea soouavacestediaaeaceens tateeesdeahacateaaevenetts 
Headteacher Email address ........cccccesesessescsessssescescaecaeseesescsecueceeseseeceesesseesesaesesecassesseeaeaesecaeeeeacseeecaeees 
Name of School Contact (if Not headteacher)..........cscsccssecsecsecsecseceesessesecesceeceecseceseessesaesseeeeseeeneeas 
School Contact Ema il:adareSs: sisiccsiscccéscsceceszs tds secauzsceneysbedectaeldbanete Yan ctearesde ds cvcets cantanete cieapsbevecchatiencdivcs 


Signature Of Neadteachersisscccsicsss ciidsvsccieisslssastecessdeeazeadsdvniesdcvaddesess salesd catteds DAU C sted caiciscatesidaiesiodies 


Thank you for agreeing to take part in this research. Please return this consent form by post to: 
Mrs D Parsons, Discover Children’s Story Centre, 383-387 High Street, Stratford, London, 
E15 4QZ 
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Appendix 6: Parent and pupil information letter 


AB | Weation e Children’s Story C 
THE UNIVERSITY Of York “WDurham “= =wsounent | IS COVEL 


Foundation 


Evaluation of the Discover Literacy Summer School intervention: 


Information Letter for Parents 
[INSERT DATE] 
[INSERT SCHOOL NAME] 
Dear Parent / Carer 


Durham University and the University of York have been asked by the Education Endowment Foundation (an 
organisation funding research into education) to independently evaluate the Discover Literacy Summer School 
programme. The summer school is being led by the Discover Children’s Story Centre (a charity organisation in 
the local area). 


Your child’s school has been chosen to take part in the evaluation and have identified your child as suitable for 
the study. We would like to invite your child to take part. 


What is the purpose of the evaluation? 


The Discover Literacy Summer School programme is designed to improve children’s writing skills during the 
transition from Year 6 to Year 7. Good writing skills are important for all children. This evaluation will find out how 
helpful the Discover Literacy Summer School is. The results of this study could influence the provision of literacy 
summer schools such as this in the future. 


What does this mean for my child? 


To find out how well the summer school programme works, children whose parents consent for them to take part 
will be allocated completely at random by computer, to either attend a four-week summer school or to receive a 
school-based literacy session delivered on a Saturday morning in the autumn term of Year 7. Both activities are 
offered completely free of charge and further details are provided below. All schools will continue to teach 
children writing skills during term time. 


The summer school 


Children allocated to the summer school group will be invited to attend a four-week literacy summer school. This 
will be held at Lister Community School from 29" July to 23° August 2013, Mondays to Fridays from 8.30am and 
4pm. The programme will combine literacy sessions with a range of exciting and memorable activities, such as 
trips to West End performances, opportunities to meet professional writers and poets and sporting activities. 
Further details of the programme will be sent to you, should you wish your child to take part and they are 
allocated to this group. 


Autumn term Saturday literacy sessions 
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Children allocated to this group will receive a half-day Saturday session in the autumn 2013 term, consisting of an 
intensive graphic novel/comic writing activity, led by a Discover artist and Story Builder. Each child will be given a 
complimentary book from Discover, signed by the author. Further details of these autumn sessions will be sent to 
you, should you wish your child to take part and they are allocated to this group. 


What will happen next? 


If you consent for your child to take part we will collect information about your child from your child’s primary 
school, and later from your child’s secondary school. Your child’s school will provide information including your 
child’s name, date of birth, gender, unique pupil number, details on your child’s current National Curriculum 
writing level, free school meal status, eligibility for pupil premium and English as an Additional Language (EAL) 
status. 


In order to measure the effectiveness of the summer school intervention, both groups of pupils will be tested at 
the start of the autumn term 2013 so that we can estimate what effect the intervention has had on their writing 
skills. We expect that your child will enjoy doing the tests and being part of the programme. 


Your child’s information will be treated with the strictest confidence. Named data will be matched with the National 
Pupil Database and shared between the evaluation team, Discover charity and EEF. We will not use your child’s 
name or the name of the school in any report arising from the research. Your child’s information will be kept 
confidential at all times. Your child may withdraw from the evaluation at any time. 


Thank you for reading this information. If you are happy for your child to take part in this study, please 
complete the enclosed consent form and return it to your child’s school by [INSERT DATE]. 


If you would like further information about the Discover Literacy Summer School evaluation please contact Laura 
Jefferson, the Evaluation Coordinator: laura.jefferson@york.ac.uk ; 01904 321511 


Yours faithfully 


Professor David Torgerson (University of York) 
Professor Carole Torgerson (Durham University) 
Discover Children’s Story Centre 


Education Endowment Foundation 
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Appendix 7: Parent and pupil consent form 


Evaluation of the Discover Literacy Summer School intervention: 


Parent Consent Form 


If you would like your child to take part in the Discover Literacy Summary School programme 
evaluation, please complete the following information and return this form to your child’s school as 
soon as possible. 


Please initial in boxes 


1. |AGREE to my child taking part in the Discover Literacy Summer School evaluation 


randomly allocated to attend either the four-week summer school or an autumn 
term literacy session as part of this evaluation 


3. |AGREE to my child undertaking a test of their writing ability in the autumn term 
2013 


4. | AGREE to my child’s data being shared for use in the Discover Literacy Summer 
School evaluation 


2. |understand that there are two programmes and AGREE to my child being | 


Parent/Carer Sign ature......c.ccccccccssssscescsesssscssescsececscsesecsececsesceecaeseaessescseeneseeeecas Dats. 2.2525. fscancdevesvectenie sais 
GIS NAIM iesieen cas ccasediacedacaasstesvencsascaice saat lvececcuedculcssaeucuestaslcavaneasueatecdevadacevagsarvaetawdnead osstes era teatasdencs seduedesousatesedanets 
HOMO address OF CHINE. seseccectucastissesesenstassestecrennpasencnliennvcanetbuavevetersenesexvh usa veasdesensoezslaelaed tanrsbvhorsebavsuessasbanetaye 
Email address:Of parent? sxcccse.ceied sececskstes even cesarean ates otadaclactede Ak a tdaeeteatieis eect tases eee: 
HOME: CONTACE NUMIDENS fescsocedaicccetects sos leeats abn ctasexsecsdea ctu simsaedeldeddeesbapnsaiadcaudeec¥enveasderes sosuaa suasnaansaceustawwanbaeeloudetlevhe 
Mobile:contact Number? c...scccsccc-cccesdejcscestevssetsacsvessacedvelendsaviiedsiccteyicdenenesecceuesstudasdnedsotavsietaviveten uareueesdeveeees 
Ghild’s Pritrary SCHOO 4: tse) ssccvceses vasedsecsesheseeetortecdavesegscchateshataesbetdstegi chad sisiggerebbeswualaseie feapededacseveubastennsteseanteasennens 
Child's primary SChOOlClass:23secceedoccsc ciedeciees ns hertecetetieeesctesiaret ccestuatetr cies rie eae th heieeeeeeeeteelt 
Child’s S@CONAArY SCHOOI.......c.cescssssecsesessescecsesscssescseseuscseseceecassesascsesesscsacaesesassesesssessesaesaeseeessssnseeeseesasassaeasensesens 
Child’s secondary school class (if KNOWN).......cscsssssssssssseecsesessescescsesesaeseeecsesesseseeesassesaesseeceesesaeseeesaesessesasesensees 


PLEASE RETURN THIS FORM TO YOUR CHILD’S CLASS TEACHER by: [insert date] 
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Appendix 8: Observation Information Leaflet 


ae a A Education e Children’s Story Centre 
THE UNIVERSITY OfYork Wena Essanniant DISCOV ver 


Foundation 


Discover Children’s Story Centre summer writing workshop: 
Observation Information Sheet 


The Education Endowment Foundation has asked researchers at the University of 
York and Durham University to evaluate the Discover Children’s Story Centre 
summer writing workshop intervention being delivered and supported by the 
Discover Children’s Story Centre in 2013. 


As part of this evaluation we will be holding observations of the Discover Children’s 
Story Centre summer writing workshop being delivered to a class of pupils. We will 
use the observation to help us better understand the Discover Children’s Story 
Centre summer writing workshops and evaluate its use. Neither yourself nor any 
pupils will be identified in any reports resulting from the evaluation. 


It would be helpful to us if we could observe you delivering the Discover Children’s 
Story Centre summer writing workshop to a class of pupils but you do not have to 
agree to be observed. If you decide you do want to take part then change your mind, 
you are free to withdraw at any time, without giving a reason. 


If you decide you would like to take part, please sign and return the consent form to 
the researcher. 


If you would like any more information please contact Dr Laura Jefferson. 
Tel: 01904 321511 


Email: laura.jefferson@york.ac.uk 


Thank you for considering taking part 
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Appendix 9: Observation Consent Form 


Education e Children’s Story Centre 


ae A 
THE UNIVERSITY Of York we -vsownen  YIGCOV ver 


Discover Children’s Story Centre summer writing workshop: 


Observation Consent Form 


Please initial 
each box 
1. | confirm that | have read and understood the Discover Children’s 
Story Centre summer writing workshop: Observation Information | | 
Sheet. | have had the opportunity to consider the information and 
ask questions. 


2. | understand that my participation is voluntary and that | am free to 
withdraw at any time, without giving any reason. 


working on the evaluation and will be stored securely. | understand 
that neither I, nor my school, nor any pupils will be identified in any 
reports resulting from the evaluation. 


4. | agree to be observed delivering a session of the Discover 
Children’s Story Centre summer writing workshop intervention to a 
class of pupils. 


3. | understand that my details will be provided to researchers | | 


Teacher Namie (PleaSe PAU ecsaca scien cacanensiatsscsncsesaecostavessesdastecasasensorelgslaionwasierausanimaasoes ses 


Te ache SIS Mat UC eccccsts secesdeieesssdicsessacecvecde ctsdecasesenseaiscasacedhaeseateaioniaanteieenaiscadesen teaeiainesieatserenae wank 


Thank you for taking part 


No) ef=\ate | fer-t<) 


Appendix 10: Missing data exploration 


Table 5: Completion of short and long parts of test and primary outcome by school 


The short form part of the test consists of exercises 1, 2, 3 and 4. 


Complete Partial Missing | Complete’ Partial Missing Complete Partial Missing 
10 2 1 0 1 1 0 1 1 0 1 
11 17 6 1 10 7 0 10 1 6 10 
12 18 0 13 5 12 4 2 0 15 3 
13 15 13 0 2 13 0 2 9 4 2 
14 3 0 2 1 1 2 0 0 3 0 
15 13 0 0 13 2 2 9 0 2 11 
16 4 1 0 3 3 0 1 1 2 1 
17 2 2 0 0 2 0 0 2 0 0 
18 7 0 0 7 0 6 1 0 6 1 
19 3 0 0 3 3 0 0 0 3 0 
20 1 0 0 1 1 0 0 0 1 0 
21 4 0 0 4 4 0 0 0 0 4 
22 7 6 0 1 7 0 0 6 0 1 
23 28 5 3 20 9 0 19 4 4 20 
Total 124 34 19 71 65 14 45 24 46 54 
27.4% 15.3% 57.3% 52.4% 11.3% 36.3% 19.4% 37.1% 43.5% 


The long form part of the test consists of exercises 3x, 4x, 5 and 6. 


The primary outcome is a subset of the long form parts of the test (exercises 5 and 6). This means 
that a pupil may have fully completed the primary outcome but not fully completed the long form of the 
test (e.g. if they did not answer exercise 3x and/or exercise 4x). 


Table 6: Number of returned and missing test papers, by school. 


Total Intervention Control Total Intervention Control 
10 2 1 0 1 1 1 0 
11 17 7 5 2 10 4 6 
12 18 16 11 5 2 0 2 
13 15 13 6 7 2 2 0 
14 3 3 3 0 0 0 0 
15 13 4 4 0 9 4 5 
16 4 3 1 2 1 1 0 
17 2 2 1 1 0 0 0 
18 i 6 4 2 1 il 0 
19 3 3 2 1 0 0 0 
20 1 1 1 0 0 0 0 
21 4 4 3 1 0 0 0 
22 7 7 4 3 0 0 0 
23 28 9 5 4 19 12 7 
TOTAL 124 79 50 29 45 25 20 
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Table 7: Completion of test by exercise 


1 72 52 
2 74 50 
3 78 46 
4 75 49 
3x 49 75 
4x 57 67 
5 53 71 
6 34 90 
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Appendix 11. Security rating summary 


1.Design: ange 3. Attrition: 4. Balance: Saree 
What is the ae What is the Adjustment validity: 
minimum PNol[Urident-laimne) 


lity of th ENV me) me | to rating t : 
eon eeliobi detectable creeecats patdobid Jee account for issues 
design of the out from the Fleaeeleaimiels : 
effect at the with 


evaluation? evaluation? balance? : : 
start? Tala-g)a-1¢-14() arg 


Evaluation design Implementation Analysis and interpretation 


aaaaa < 10% Well-balanced No threats to 
observables validity 


Fair and clear 
4688 experimental design < 20% 
RCT, RDD 


Well-matched 


comparison (quasi- < 30% 
experiment) 


(quasi-experiment) 
Comparison group with 5 
= poor or no matching < 50% 
No comparator > 0.6 yp9) | imbatanced on 
observales 


The final security rating for this trial is of. This means that the evaluation has very low security and 
adds little or nothing to the existing evidence base. The trial was designed as a randomised, 


independently-run, efficacy trial and was powered to achieve a maximum of 3 


The trial had very high attrition of over 70%. The evaluators used regression methods to try and 
control for observable characteristics, but this could not account for the potential imbalance on 
unobservable characteristics. In addition there were significant threats to the validity of the findings. 
Schools delivered the tests, rather than blind by evaluators. 


Additional detail about the EEF’s’ security rating system can be found at: 
www.educationendowmentfoundation.org.uk/evaluation. 


You may re-use this document/publication (not including logos) free of charge in any format or 
medium, under the terms of the Open Government Licence v2.0. 


To view this licence, visit www.nationalarchives.gov.uk/doc/open-government-licence/version/2 
or email: psi@nationalarchives.gsi.gov.uk 


Where we have identified any third party copyright information you will need to obtain permission from 
the copyright holders concerned. The views expressed in this report are the authors’ and do not 
necessarily reflect those of the Department for Education. 


This document is available for download at www.educationendowmentfoundation.org.uk. 


A Education 


Endowment 
Foundation 


The Education Endowment Foundation 

9th Floor, Millbank Tower 

21-24 Millbank 

London 

SW1P 4QP 
www.educationendowmentfoundation.org.uk 


